Filter limit??

Hi all,

I use a filter to exclude objects from an export operation. It's a big filter, and recently (today) it got even bigger and suddenly I'm getting a stack overflow error.

The filter is done as an include file, using the format:

Filter f1 = (attribute "Absolute Number" != "221" // -
&& attribute "Absolute Number" != "224" && attribute "Absolute Number" != "227" && attribute "Absolute Number" != "228" // -
....

The include file has (currently) 522 rows, and I don't know how many ID's (it varies, but typically maybe 5 or 6 ID's per row).

Is there a limit, and can it be changed, for how big a filter definition can be?

Thanks in advance!

-SteveK.
SteveK01 - Mon May 10 20:34:12 EDT 2010

Re: Filter limit??
Mathias Mamsch - Tue May 11 12:33:42 EDT 2010

I think the problem might not be the filter definition but the concatenation of all those filters. Remember when you do:

a + b + c

then DOORS will do (b+c), store the result on the stack, then do a + (result on stack) ... When you have 500x5 concatenations like this:
(attribute "XY") != "277" which will put the result of attribute "XY" on the stack, then evaluate the != to a filter and put this filter on the stack, then you have about 500x5x2 = 5000 items on the stack which is bigger than the default stack size of 4096.

You can either increase the stack size to a higher number (e.g. using "pragma stack, 50000") or you could do your include file like this:

Filter f1 = (attribute "Absolute Number" != "123")
f1 &= (attribute "Absolute Number" != "124")
// ...

If my suspicion is right, the error should go away in both cases.

Regards, Mathias

Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS

Re: Filter limit??
Mathias Mamsch - Tue May 11 12:33:52 EDT 2010

Filter f1 = (attribute "Absolute Number" != "123")
f1 = f1 && (attribute "Absolute Number" != "124")
// ...

If my suspicion is right, the error should go away in both cases.

Regards, Mathias

Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS

Re: Filter limit??
SteveK01 - Tue May 11 12:56:54 EDT 2010

Hi Mathias,

Thanks so much. I had already opened up the stack, and it didn't help. But splitting the filter into two parts and then joining them back together works nicely. Simple solution, but I was basically brain dead at that point and couldn't see the solution. Works great now.

Thanks again!
-Steve

Re: Filter limit??
kabr - Thu May 12 04:00:15 EDT 2011

Hi all,

I would like to come back to this topic because of its title.

Does anybody know if there is a limit for the size of a filter string, or for the number of simple filter expressions to be combined to a complex filter expression?

Karl

Re: Filter limit??
Mathias Mamsch - Thu May 12 06:30:00 EDT 2011

kabr - Thu May 12 04:00:15 EDT 2011
Hi all,

I would like to come back to this topic because of its title.

Does anybody know if there is a limit for the size of a filter string, or for the number of simple filter expressions to be combined to a complex filter expression?

Karl

In fact I don't know, but it should not be hard to test...

pragma runLim, 0
int maxSize = 1000
 
Filter F
string errMsg
Module m = null
if (!null (errMsg = create("Filter Test Module", "", "PREF_", 1, m) ) || null m) error "Cannot create module: " errMsg "\n"
current = m
showExplorer false
 
int curSize; for curSize in 1:maxSize do {
   create m 
  
   print "Checking for " curSize " elements ... "
 
   if (curSize == 1) {
    F = attribute "Absolute Number" == curSize ""
   } else {
      F = F or attribute "Absolute Number" == curSize ""
   }
 
   int timeStart = getTickCount_
   set F; filtering on;
   int timeEnd = getTickCount_ - timeStart
 
   iCount = 0; Object o; for o in m do iCount ++
   
   print "Filtered " iCount " elements in " timeEnd " ms."
   if (iCount == curSize) print "OK.\n" else print "### NOT OK ###\n"
}
 
downgrade m; close m
 
softDelete module "Filter Test Module"
hardDelete module "Filter Test Module"

I tested up to a filter size to 1000 which gave me:

Checking for 1 elements ... Filtered 1 elements in 0 ms.OK.
Checking for 2 elements ... Filtered 2 elements in 0 ms.OK.
Checking for 3 elements ... Filtered 3 elements in 15 ms.OK.
...
Checking for 649 elements ... Filtered 649 elements in 125 ms.OK.
Checking for 650 elements ... Filtered 650 elements in 140 ms.OK.
Checking for 651 elements ... Filtered 651 elements in 141 ms.OK.
...
Checking for 998 elements ... Filtered 998 elements in 250 ms.OK.
Checking for 999 elements ... Filtered 999 elements in 250 ms.OK.
Checking for 1000 elements ... Filtered 1000 elements in 266 ms.OK.

Feel free to check more ... If speed is of concern you might want to use accept() or reject() for creating a filter from DXL instead of the concatenation... Hope this answer helps, regards, Mathias

Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS

Re: Filter limit??
kabr - Thu May 12 08:00:07 EDT 2011

Mathias Mamsch - Thu May 12 06:30:00 EDT 2011

In fact I don't know, but it should not be hard to test...

pragma runLim, 0
int maxSize = 1000
 
Filter F
string errMsg
Module m = null
if (!null (errMsg = create("Filter Test Module", "", "PREF_", 1, m) ) || null m) error "Cannot create module: " errMsg "\n"
current = m
showExplorer false
 
int curSize; for curSize in 1:maxSize do {
   create m 
  
   print "Checking for " curSize " elements ... "
 
   if (curSize == 1) {
    F = attribute "Absolute Number" == curSize ""
   } else {
      F = F or attribute "Absolute Number" == curSize ""
   }
 
   int timeStart = getTickCount_
   set F; filtering on;
   int timeEnd = getTickCount_ - timeStart
 
   iCount = 0; Object o; for o in m do iCount ++
   
   print "Filtered " iCount " elements in " timeEnd " ms."
   if (iCount == curSize) print "OK.\n" else print "### NOT OK ###\n"
}
 
downgrade m; close m
 
softDelete module "Filter Test Module"
hardDelete module "Filter Test Module"

I tested up to a filter size to 1000 which gave me:

Checking for 1 elements ... Filtered 1 elements in 0 ms.OK.
Checking for 2 elements ... Filtered 2 elements in 0 ms.OK.
Checking for 3 elements ... Filtered 3 elements in 15 ms.OK.
...
Checking for 649 elements ... Filtered 649 elements in 125 ms.OK.
Checking for 650 elements ... Filtered 650 elements in 140 ms.OK.
Checking for 651 elements ... Filtered 651 elements in 141 ms.OK.
...
Checking for 998 elements ... Filtered 998 elements in 250 ms.OK.
Checking for 999 elements ... Filtered 999 elements in 250 ms.OK.
Checking for 1000 elements ... Filtered 1000 elements in 266 ms.OK.

Feel free to check more ... If speed is of concern you might want to use accept() or reject() for creating a filter from DXL instead of the concatenation... Hope this answer helps, regards, Mathias

Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS

Thanks Mathias,

this is basically what I wanted to know.
At least I now know that in DXL I can build a filter that big and apply it.
From the filter dialog this does not seem to work.
I have one user who tried to build a filter with 72 criteria and DOORS refused to compute the result.

And yes, for my own purposes I would probably use accept and reject in a DXL script but users do not have the rights to edit DXL in our database.

rgds
Karl

Re: Filter limit??
Mathias Mamsch - Thu May 12 08:22:30 EDT 2011

kabr - Thu May 12 08:00:07 EDT 2011
Thanks Mathias,

this is basically what I wanted to know.
At least I now know that in DXL I can build a filter that big and apply it.
From the filter dialog this does not seem to work.
I have one user who tried to build a filter with 72 criteria and DOORS refused to compute the result.

And yes, for my own purposes I would probably use accept and reject in a DXL script but users do not have the rights to edit DXL in our database.

rgds
Karl

Well I am not sure if this example is really valid for ALL kind of filters. For example, I only used 'Attribute ... == value' filters, there might be restrictions for other filter kinds. If one of your users says that he has problems with a big filter, maybe a little details for testing the specific filter would be good. And the way DOORS 'refuses' to calculate the result. Will it just calculate a seemingly wrong result (could be a 'PEBCAK', for a filter that big), or will DOORS crash or did he get a message, that the filter is too big?

By the way there is another flaw in the test script. Since the objects are created one by one and the filter is actually trying to show ALL objects, there could be a possibility where the filter would just not work at all (all objects are shown) and still the program would show "ok". So you might want to move the object creation loop outside of the filter loop.

Regards, Mathias

Mathias Mamsch, IT-QBase GmbH, Consultant for Requirement Engineering and D00RS